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ABSTRACT 

This report describes a collection of procedures, 
with illustrative examples , for selecting and portraying 
microcomputer courseware in a manner that enables others to make 
their own judgments of courseware quality. Following a discussion of 
perspective and a report outline, section 3 deals with assumptions 
underlying the search to identify specific aspects of courseware form 
to illuminate. A fourth section describes the data sets selected as 
most relevant for the portrayal of courseware structure, and a 
rationale for the purpose of courseware structure portrayal and for 
specific portrayal content is provided in section 5. The sixth 
section presents three illustrative portrayal strategies for 
courseware criticism, including a mastery model analysis comparative 
table; a narrative of student courseware use with a flow chart of 
inferred courseware structure; and brief summary flow charts of two 
student paths and a simplified, .inferred, author-designed path. 
Section 7 outlines a proposal for testing the effects of different 
kinds of portrayal, while section 8 presents an example of the use of 
an open-ended evaluation check list for conducting microcomputer 
courseware criticism. Appendices provide guidelines for structured 
observation of student paths through microcomputer courseware and a 
copy of the evaluation checklist with instructions included. A 
15-item bibliography concludes the report. (ESR) 
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PREFACE 



The Research on Evaluation Program is a Northwest Regional 
Educational Laboratory project of research, development, testing, 
and training designed to create new evaluation methodologies for 
use in education. This document is one of a series of papers 
and reports produced by program staff* visiting scholars, adjunct 
scholars, and project collaborators—all members of a cooperative 
network of colleagues working on the development of new 
methodologies. 

How can one evaluate microcomputer courseware in a way that 
helps others make their own judgments of its quality? What 
alternative styles of evaluative presentations are available? 
This report answers these and related questions by describing a 
collection of procedures, with illustrative examples, for 
salecting and portraying evaluative information, A checklist 
for conducting microcomputer courseware criticism is also 
Included. 



Nick L, Smith, Editor 
Paper and Report Series 
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MAKING COURSEWARE TRANSPARENT: 
BEYOND INITIAL SCREENING 



Gabriel Della-P1ana 
Connie Kubo Della-Piana 

University of Utah 

It 1s argued elsewhere that the kind of criticism most needed 
today 1s that which helps us to see how things work (Della-P1ana, 1931a). 
The possibilities of applying that perspective to meta-evaluatlon and 
the evaluation of microcomputer-based Instruction were sketched out In 
previous reports (Della-P1ana» 1981b, 1981c)*. In the present report 
we briefly restate the perspective of evaluation as "making a work 
transparent" and outline more clearly a methodology for gathering and 
portraying data within such a framework, with special application to 
criticism of microcomputer-based Instruction. 

1 .0 The Perspective 

The kind of criticism that our perspective opposes is that which 
gives us the definitive Interpretation as to what a piece of courseware 
1s about or makes the final judgment in the sense of pronouncing the 
courseware as good or bad fdr us. To give a final Interpretation, 
paraphrase, or summary judgment of a work puts an end to 1t, It does 
the thinking and feeling for the user. It makes the user's response 
unnecessary* It may even, 1n the long run, make the user helpless or 
dependent upon others for critical Interpretation and judgment. As for the 
courseware developer or distributor, it may produce defensiveness or 
complacency with respect to the need for revision. Our perspective is 



that the function of criticism is to show how things work, to make them 
transparent, to demystify. But it is not enough to show how things work. 
The portrayal of the workings of a piece of courseware must have utility 
for two audiences. It must involve users in making their own interpre- 
tations* judgments and adaptations of the courseware for their own 
purposes. It must involve developsrs and distributors in "revision/ 1 
in seeing the work again or looking back at it from a perspective that 
may drive redevelopment. 

2.0 Outline of the Report 

The remainder of this report will proceed as follows. Section 3*0 
deals with assumptions underlying the search for which aspects of form 
to illuminate. Specifically* we will be concerned with the Implications 
of Information processing capacity* the state of the art on how to 
design Instruction* and the utility of different kinds of data as guides 
to what kind of Information to gather. In Section 4.0 we outline the 
data sets we have selected as most relevant to the portrayal of the 
structure of courseware. In Section 5,0 we provide a rationale for the 
purpose of portrayal of courseware structure and the content of portrayals. 
The emphasis on purpose is on how portrayal nay serve the audiences to 
which 1t is directed. The emphasis on content is on how juxtapositions 
of data provide dissonance that involves readers in making their own 
interpretations and judgments. Section 6.0 presents three illustrative 
portrayal strategies for courseware criticism* Section 7.0 outlines 
how we would propose to test the effects of different kinds of portrayal 
and, finally* Section 8*0 presents an "open-ended checklist" consistent 
with our courseware criticism perspective along with an example of 
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its use* The checklist is an attempt to provide a simpler application of 
our perspective that goes beyond the popular checklists » but not so much 
in depth as the methodology we propose in previous sections* 

3.0 Assumptions Underlying the Search for What to Illuminate 

If we are to make courseware transparent, we must decide on what 
in the courseware one might best illuminate for the purposes noted above* 
But where shall we find a framework to guide us in selecting what is to 
be portrayed? There is certainly no agreed-upon technology of instruction* 
Computer hardware is changing rapidly, system components are changing 
£e*g*, video discs and interactive operation of microcomputers with larger 
time-sharing systems, greater capacity of microcomputers) , authoring 
systems are primitive and likely to change rapidiy, and courseware itself 
has not even caught up with present possibilities* Indeed, the contexts 
in which courseware will be use! for jobs and their social, political and 
economic contexts, are relatively unknown to us* Thus, protrayal of 
courseware structures of today will likely be irrelevant to the courseware 
of the near future and any particular Instance of courseware criticism 
could easily appear irrelevant except for historic analysis or as a 
source of ideas or strategies to avoid or pursue further. 

Given the rapidly changing form of computer hardware, software, 
authoring systems, courseware and contexts of use, w e have sought a 
set of guiding assumptions that might lead us to focus on data that 
illuminate the structure of courseware in ways that have Immediate 
utility and some long-term value* The following six assumptions seem 
to us to be appropriate guides* 

First, the potential user of courseware is an information proces- 
sing system of limited capacity (Simon, 1971, pp* 40, 41, 62)* In other 
words, there are limits to the amount of data or number of variables 



that a person can efficiently attend to. This suggests the need to 
search for an appropriately limited and useful subset of courseware 
structures for the user to grasp and process, We need to do more than 
"spew out data in forms which human beings then have to process exten- 
sively.* 

Second, instructional prescriptions derived from cognitive, 
developmental or behavioral psychology must be viewed as testable 
contemporary hunches rather than as procedural laws. Thus, prescrip- 
tions may be tentative guides to instructional design and development, 
but they are best seen as hypotheses concerning instruction derived from 
constructs and thus as interpretations that must be validated in many 
contexts, A prescriptive science of design does not appear to be at 
hand (Calfee, 1981; GUser, 1982; Resnick, 1981), 

Third, the perhaps most valuable data set against which all others 
may be usefully juxtaposed is a detailed description of the cognitive 
processes of the student working through an instructional sequence 
(Resnick, 1981, p, 693)* This kind of information has immediate practi- 
cal utility for diagnosis or adaptation of instruction to individual 
differences as well as long-range utility for generating significant 
instructional prescriptions. 

Fourth, any reasonably permanent set of criteria for courseware 
criticism must derive from the experience of application to many 
instructional packages in many subject domains. It is only upon the 
base of numerous case studies that critical criteria and procedures may 
be refined and primarily then that it would be useful to judge the 
criteria and procedures against critical competitors. 



Fifth, the portrayal of the structure of a specific piece of 
courseware will vary for different audiences. In our work, the portrayal 
will always be for the purpose of helping the user and the developer/ 
distributor. The focus for users will be on helping them make their 
own interpretations and judgments about Instructional utility for their 
own purposes or designing ways of modifying or using courseware already 
selected. For the developer/distributor the focus will be on helping 
them to look again at their courseware for purposes of redevelopment 
or for modifying their representations as to how the courseware accom- 
plishes what it does accomplish or what might be accomplished (Oella-Piana, 
1981, pp, 211-246; Smith, 1982, pp, 177-304). 

Sixth, courseware criticism will typically be based in part on a 
form of naturalistic inquiry and must be subject to the usual canons of 
such inquiry (Guba and Lincoln, 1981, pp. 85*127), In addition, it must 
specifically attend to the tnterdependency of such Instruction with other 
systems surrounding it (Sronfenbrenner, 1979), the characteristics of 
learners, economy or worth in relation to cost, and general 1zab1l ity 
of performance across different contexts (Gilbert, 1978, pp. 284-286)* 

4,0 Oata Sets to be Illuminated 

Figure 1,0 presents graphically the data sets that we see as most 
relevant to the portrayal of the structure of courseware consistent 
with the above assumptions. In this section we provide a ceneral 
orientation to those data sets and to the methodology for gathering 
data. 

The author-defined domain of the courseware is derived from three 
data sources. The manual or courseware Itself may literally specify 
objectives or Intents of the author as to what the courseware is 
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designed to accomplish. Some author objectives may be inferred from an 
examination of the instructional strategy and content. Other author 
objectives may be inferred from an analysis of the final mastery test 
for the course, if there is one, or from submastery tests. The author- 
defined domain is what is designed to happen as the student goes through 
the courseware and completes it. 

The student-defined domain of the courseware is derived from three 
data sources. When we say "student-defined domain 11 we mean to suggest 
that what the course is may be defined in part by what the student does 
in going through the course or upon completion of the course and 
presumably as a result of the course. The Instructional path of the 
student in using the manual, in using other supportive material* and 1n 
interaction with the computer, defines what the course 1s for a student* 
Likewise, when the student seeks or gets assistance from live Interaction 
with a teacher, a1de» or peers, this too helps to define the actual 
course for a student* And the student's performance on submastery or 
final mastery tests in relation to his/her aptitude also defines what the 
course is or was for the student* The student-defined domain 1s what 
does happen as the student goes through a course and completes it. 

The ideal-defined domain of the courseware is also derived from 
three sources of data* When we say "ideal-defined domain" we mean to 
suggest what the course might be given some specific set of user assump- 
tions, "Valued accomplishments" related to the course may be determined 
by some rational consensual procedure applied to policy-shaping members 
of a community (subject-matter experts, on-line administrators, patrons 
of the school). This wo^d help define the "Ideal" course. However, 
the number of accomplishments some policy-shaping group might hope for 
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Is likely to be beyond what could reasonably be achieved* Thus, such a 
list may be reduced by empirical test or informed hunches as to which 
subset of valued accomplishments one might "teach-towards" that would, 
as a by-product, also get the largest number of other valued accomplish* 
ments* For example, If one taught "reading comprehension' 1 without any 
written work, the student would not learn to write* But if one taught 
writing in certain ways, one would also likely get accomplishments in 
reading comprehension* Finally, a list of valued accomplishments may 
be reduced further by de-selection of those for which there Is the 
least favorable ratio of accomplishments to costs of instruction* The 
ideal-defined domain is thus what should happen as the student goes 
through the course and completes it* 

Overlapping data sets are indicated in part by areas A, B, C, 0, in 

Figure 1*0* It is the points of consistency and inconsistency both 

* 

within a data set (e*g*, author-defined objectives literally stated 
versus those inferred from final mastery tests or instructional strategy) 
and among data sets (areas labelled A, B, C, 0 in Figure 1 *Q) that should 
be the focus for portrayals of what courseware accomplishes and how it 
accomplishes what it does* 

The number of possible overlapping data sets is of course quite 
large* Thus, in practice, one must choose which juxtapositions to 
portray* At this stage of our thinking on the matter, we believe that 
the choice will be largely subjective with an eye to portrayal of data 
which appears most likely to achieve the objectives suggested in 
Section 1*0, "The Perspective," and in Section 2*0 on "Assumptions 
Underlying the Search for What to Illuminate*" 



Objectives inferred from 
analysis of Instructional 
strategy and content 



Objectives specified 
In manual or 
courseware 




Author* defined domain 
of the courseware (what 
Is designed to happen) 




Objectives inferred 
from analysis of final 
mastery tests or sub* 
mastery tests 




Ideal-deffned domain 
of the courseware 
{what should happen) 



Area 

C 



Student-defined 
dona In of the 
courseware (what 
does happen) 




Valued 

a c com pi Ishnents 



Economy or worth as a 
ratio of valued accom* 
pUshmefits to costs of 
Instruction 



Generalization power of 
valued accomplishments 



Instructional 
Path of student 
in relation to 
manual and 
computer 

Student performance 
on submastery tests, 
final mastery test, 
and aptitude measures 



Student interaction with 
teacher, aides* peerSt or 
others 



FIGURE 1.0. A graphic representation of nine sources of data for making 
microcomputer courseware transparent. It is the jux:aposi- 
tlon of data represented by areas A, S, C, and 0 that is 
the focus for portrayal of courseware. 
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Procedures for Obtaining Data are presented in summary form in 
Table KO for each of the data sources relevant to the three vantage 
points for defining the domain of the courseware. A description of all 
these procedures 1s beyond the scope of the present report. One set of 
procedures (for structured observation of student path through the 
courseware) Is presented In some detail In Appendix A. Other procedures 
will be Illustrated In later sections* but not presented in detail. Any 
competent evaluator knows how to gather such data. 

An examination of the summary of procedures in Table 1.0 suggests 
that the kind of criticism proposed Is based on data that must be 
gathered and analyzed under the direction of personnel with competency 
In cognitive process analysis* knowledge of the subject matter of the 
courseware, knowledge of Instructional design process* and skill in 
evaluation of Instruction. Naturally* this would ordinarily require a 
team of critics. This Is a big order and such Intensive criticism 
should certainly be limited to courseware that Is worth the attention 
because of being exemplary in some Important ways* However* In a later 
section* we will also propose a "checklist approach 11 consistent with 
our perspective. The checklist Is designed for less time-consuming 
criticism and for more general use consistent with the perspective of 
the more In-depth criticism, 

5,0 Data Portrayal: Purpose and Content 

How one presents the results of one's critical analysis should 
take into account the purpose of the portrayal in relation to specific 
audiences. In this section* we outline briefly some purposes we see 
as appropriate and some Juxtapositions of Information relevant to these 
purposes , 



TABLE 1.0 

DATA GATHERING PROCEDURES APPROPRIATE TO THREE PERSPECTIVES FOR DEFINING A COURSEWARE PACKAGE 



Oomaln Definition Strategies 



Data Sources for Each Domain Category 


Procedures for Obtaining Oata 


Author-Deft ned Domain 

1. Objectives specified In courseware, 
manual or other supportive material. 

2. Objectives Inferred from analysis of 
Instructional strategy. 

3. Objectives Inferred from analysis of 
final mastery test. 


a) Collection of objectives stated within the courseware (on the 
computer monitor) or In other supportive material. The objec- 
tives may be reorganized according to some system for formal 
specification of objectives. 

b) E valuator- Inferred objectives as determined by going through the 
"course* 1 as a student and no+ing what the instructional strate- 
gies appear to teach. 

c) E valuator- Inferred objectives as determined by evaluator analysis 
of final mastery test or submastery tests. 


Student-Defined Domain 

4. Instructional path of student In 
relation to computer Interaction 
and supportive material. 

5. Student performance on submastery 
and final mastery tests as function 
of aptitude. 

6. Student Interaction with teachers, 
aides, peers or "noncourse" sources 
of Information. 


d) Hard-copy printout of student responses while going through 
computer-aided Instruction. 

e) Workbook (or other comparable response sheet) responses associa- 
ted with the courseware package. 

f) Structured Interviews with students after completion of instruc- 
tional segments on critical points within the Instruction. 

g) Observation of students during Instruction re. interaction with 
computers, supportive materials, peersi or others and follow-up 
Interviews of key persons. 

h) Student performance on submastery and mastery tests. 


Ideal-Defined Domain 

7. Valued courseware- related 
accomplishments as judged by 
appropriate policy-shaping 
community. 

8. Generalization power of valued 
accomplishments. 

9. Economy or worth of Instruction 
as a ratio of valued accomplish- 
ments to costs of Instruction, 


1) Judgments of value of alternative models of accomplishments as 
determined by a committee using some sort of Informed consensus 
process . 

j) Estimates of generalization power of value accomplishments based 
on empirical tryout data or Informed hunches; (does mastery 
generalize to other valued accomplishments?). 

k) Estimation of the worth of Instruction based on actual/estimated 
costs and Judged value of the accomplishments. 



Purposes for Portrayal of Courseware Criticism . In Table 2-0 
we summarize some major purposes of portrayal of courseware criticism In 
relation to specific audiences. There are, of course, other audiences 
and other portrayal purposes. Earlier in this paper (Section 1.0 and 
the sixth assumption in Section 3.0) we made clear our perspective that 
courseware criticism should illuminate how things work in a way that 
involves people in making their own interpretations and judgments. 
Table 2.0 does not in Itself make that perspective clear. Thus, we 
emphasize it here. For example, the purpose of criticism "for the buyer" 
is to guide selection of packages to "try out." It would be possible to 
list criteria, have the prospective buyer apply the criteria and come up 
with a number which can be put in a decision algorithm, such as: buy 
if your rating is eight or higher, don't buy If the rating Is three or 
lower, and get more information or examine critical competitors If the 
rating Is between four and seven. We feel that such algorithms *re use- 
fill for selling. However, for buying, the portrayal of courseware 
criticism should illuminate significant structures in the courseware 
so that the prospective buyer can see how It works and make his/her own 
Interpretations and judgments as to its utility for his/her purposes and 
contexts. 

Focus on Juxtapositions of Data Sets . We have said that our 
purpose for portrayal Is to involve people In making their own inter- 
pretations and judgments about courseware in relation to their own uses. 
One way of involving people in making their own interpretations is to 
create dissonance. And one way of creating dissonance is to Juxtapose 
data sets that are likely to create dissonance. In Figure 1.0 the 
juxtapositions we propose are represented not only by overlapping areas 
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TABLE 2-0 



SUMMARY OF PURPOSES OF PORTRAYAL OF MICROCOMPUTER 
COURSEWARE CRITICISM FOR SELECTED AUDIENCES 



Audience for the Criticism 



Purposes of the Criticism 



The buyer of Instructional packages 
(supervisor* administrator, teacher 
or school-community committee) 



Guide selection of packages to 
"try out," 



The teacher coaching, consulting 
or advising students ising the 
courseware. 



Guide adaptation of the course- 
ware or proposed utilization 
procedures for use within one's 
own context* 



The student user of the courseware, 



Motivate (by showing what can be 
accomplished) and Inform on uses and 
overcoming obstacles to use. 



Hie designer/developer/dlstrlbutor 
of the courseware package. 



Motivate to redevelop or supplement, 
or modify representations of current 
courseware* 

Motivate to change design/develop- 
ment procedures in the future. 

Motivate to modify representations 
of future courseware. 
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A, 8, C» and D, but also by juxtapositions within a data set. The number 
of possible juxtapositions of nine sources of data taken two at a time, 
regardless of order, Is thirty-six. When one Includes juxtapositions 
three at a time and also the numerous kinds of data within any given 
data source, the number of possible juxtapositions of data goes beyond 
the magnitude to which one can give serious attention. As we have 
Indicated before, we have no easy solution to the criteria for clustering 
or chunking such Information, Our preference Is to consider at least the 
thirty-six possibilities and select subjectively some subset of data 
juxtapositions to portray* To give the reader a sense of how such 
jxutaposltlons may provide dissonance that involves one in making Inter- 
pretations, we have summarized some Illustrative data juxtapositions 
In Table 3*0. The Illustrations are not Intended as proper portrayals to 
produce Involvement In the reader, but rather as examples of the kinds of 
data that could effectively be juxtaposed to obtain Invol vement: of the 
reader* 

6.0 Illustrative Strategies for Portrayal of Courseware Criticism 
In this section we provide three Illustrations of portrayal 
strategies for courseware criticism. Recent explorations of ^alternative 
evaluation strategies have been coupled with explorations In ways of 
communicating the outcomes of evaluation. For a survey of communication 
strategies In evaluation, the reader Is referred to the third volume in 
the series on 3eu Perspectives in Evaluation* edited by Nick L, Smith 
(1982)* The strategies we Illustrate here are designed to portray the 
kinds of data juxtapositions we have outlined above (see Figure 1.0 and 
Table 3,0) and to accomplish the purposes for courseware criticism also 
presented above (see Section 4.0). 
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TABLE 3.D 



ILLUSTRATIVE OA'.A SET JUXTAPOSITIONS FDR AUTHOR-DEFINED, STUDENT-DEFINED AND 
IDEAL-DEFINED DOMAINS OF MICROCOMPUTER COURSEWARE 



Juxtaposition 
(See Figure 1.0) 


Illustration jf Kinds of Information That Might Be 
In Portrayals for Selected Data Juxtapositions 


Author-Deftned (specified) 
and Author-Defined (inferred) 
and Student- Defined 


A drill and practice program Is designed bv the author to "Increase accuracy 
and rate of performance In arithmetic fundamentals through massive practice 
and immediate feedback. 4 * The Instructional strategy Includes 6DX of the 
time on any frame devoted to feedback such as fireworks flashing across the 

monitor ona tor ret t onsHvr tunr irnia t ion . n large portion or stuaents \c.uhi 

lose Interest* miss feedback* and punch in responses without much conscious 
attention. 


Student-Defined and 
Author-Defined 


A program for writing poetry with a computer Is designed by the authors so 
that students Input words tagged according to part of speech and retrieve 
by grammatical category to "write a poem" with the computer randomly selec- 
ting worus From a category. Amateur writers experiment witn wora oraer ana 
kinds of words used as Input. Inexperienced writers Input words with 
difficulty* retrieve In chaotic order* and give up after doing one **poem." 


Student-Defined (instruc- 
tional path) and Student- 
Deftnfc (performance on 
aptltid^ measures and mastery 
tests) 


A stjdent goes through a program to "teach Intermediate-grade children 
punctuation and capitalization/ 1 The student path Is as follows: the 
student reads a sentence* moves a cursor on the monitor (stopping It when 
punctuation or capitalization Is needed)* and strikes the key number 
corresponding to the appropriate response taken from alternatives at the 
bottom of the page. Students who cannot read the material with proper 
Intonation respond randomly and perform poorly on the mastery tests. 


Author-Defined and Ideal- 
Defined 


In a computer literacy program* a unit on history of computers Is designed 
by the authors to match names of computer pioneers with specific major 
accomplishments and place the accomplishments on a time line. An Ideal 
goal for history of computers would be to have students predict the effects 
on society of this new technology based on an analysis of the effects of 
past technologies and knowledge of the current situation. 


Ideal-Defined and Student- 
Defined 


A drill and practice program for basic arithmetic facts Is used by students 
who are Insecure as follows: when asked to specify number of digits and 
speed of presentation* they set standards unreallstlcally high (discouraging) 
or unreallstlcally low (no Improvement over current skills). Ideal program- 
ming puts Item difficulty and response time under student control, feut provides 
feedback on student r^pon*t: history to accommodate individual differences. 
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Mastery Model Analysis Comparative Table . This portrayal 
strategy is based on the thematic matrix analysis procedure outlined 
in Della-Piaia (1 981 , pp. 216-219)- In the present case the form is 
a comparative table . The data juxtaposition is an author-defined 
objective for part of a computer literacy program and an Ideal-defined 
objective for such a program. The portrayal is presented in Table 4,0 In 
a summary form. The first three columns of the table define a "mastery 
model 11 or objective of instruction. There are two alternative mastery 
models for comparison. The top example (A) is an "author-defined" 
model of what to teach towards and how to assess accomplishment of the 
stated objective. The model was abstracted by inference from courseware 
and accompanying material. The bottom example (B) is an "ideal-defined" 
model of what to teach towards. The ideal was derived from an analysis 
of the subject matter of social studies (Gilbert, 1976) as a source of 
predictive power. The table itself is adapted from a form designed 
also by Gilbert (1978, pp. 284*286). What stands out in the analysis 
is that the author-defined mastery model (roughly recalling important 
events in the history of computers and placing them on a time Hne) 
1s economical to assess and matches the stated general objective, 
but probably does not generalize to a more important objective (roughly, 
"using knowledge of past history of technology to predict effects of 
computer technology"). The "Ideal-defined" mastery model on the other 
hand does not match the stated general objective, Is. a perhaps more 
appropriate goal for the study of history of computers, Is expensive 
to assess, but will likely generalize to the stated general objective. 
That is, people who master the "ideal-defined" accomplishments will most 
likely also be able to perform well on the author-defined model- Thus, 
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TABLE 4.0 



Student Population: 
General Objective: 



MASTERY MODEL ANALYSIS COMPARATIVE TABLE* 
Junior High School 

Recall major events In development of computer and place them accurately (i5 yrs) 1n time, 



MlernAllve H*sltiy Hodelt 



SIluAlloft 



lit & pons* 



Art I fret 



iMorttnct 



ttntrtlliAbtllty 



Economy 
(S| (R) <M 



Choice 







OJ 




C 






OJ 




> 


OJ 




o 


4J 


1 


O 


(- 


OJ 


o 


'*-> 




jD 


4J 


O 






< 





A. Given: Mtrked Mm 
tint' fro* 1600 to 
2000 A O. list of Uy 
hlstorlCAl tvenls 
covering the period, 
Instructions: PlACt 
tw* evenls fro* eACh 
century on Ihe Ma* 
1 In* And rtcAtl And 
record compm*r his- 
tory events on Iht 
lliMt lint. 



Sflttt ^btiMl hit- 
lorHtl eventt fro* 
Mil And Accurately 

plttt t«0 I A eACh 
century, Urlle Along 
time I Int **Jor com- 
puter tvtnlt recAlltd, 



Th* dr*»ft tlM Mm 
tilth ttytrtl blttorl- 
c*l *nd cOMuttr 
history tvtnlt drown 
U 



frlMry ; Tho mi t try 
Mdtl tptclf JcttlMt 
for guiding ttutt- 
MM Mtch tnt 
ttAttd objective 
rether 



objec 



pArtlel or Incomplete 
trAntfer It expected 

tO *«.* If OM It tfc>U 

to demonttrAte thlt 
eccomplltfcMnt, on* 
It not Mcetterlly 
Able to demonttrAte 
ACCOmpllshMnt t* 
Alto, MeAChlnq 
totMroV A Mill not 
likely get you ft. 



Situation , CcoooalcAl 
to set up testing 
condition! end to prt- 
pAre the tett, 
ftesjtonse, llrltttn 
retpMse require? 
only pencil And Piper 
And It thus very 
lne*pensl ve + 
Arlj fACt, Scoring tty 
m>T difficult or 
e*penslve to develop. 
Scoring by Hind 
cuderAtely expensive. 



The question Is; Do 
you vAnt to me* sure 
And ttACh towerd too- 
thing thAt It Ineipen- 
tlvt to Accomplish, 
mitches the stAted 
objective, but dots 
not Bttch a wort 
APproPrlAlt goil of 
the study of hlstoryt 



OJ 

C iA 

•r- OJ 

**- > 

OJ -r- 

O 4J 

I O 

^— 0J 

<D +( -> 

S3 



IB) Given: turned Mm 
I Ine^fron IftOO to 
2000 M.D. list of 
letJjnologlcAl develop- 
ments Including 
conpkiter history 
tvtnls PlACtd on tht 
I lr.t I fnt . 

oJit or our* techno* 
!o jlCAl development! 
lor compjrAll *e 
Analysts with new 
Crjrvuitr technology. 
Analyze And Predict 
parallel tffects on 
tcunoetc *nd socIaI 
systems 



Select some lechno- 
ToglcAt development 
for co*PArAllv« 
AnAlysIs with compu- 
ter technology, Mrlle 
predictions of tffect 
0' tOrtpultr technology 
Along wllh MAlysIs of 
pArAllel tffects In 
other technologist tod 
rtAtons for predic- 
tion. 



Tht drAwn Hm lint 
■ Itn cooputtr hit lory 
evtnlt dYAvn In, 
llrllten PMdlcilont 
And coBptrAtlvt 
AnAlysIt of offtctt 
of coAputtr ttch- 
nology And) tOM othtr 
technologlctl dovolop- 
Mnt, 



SecondAry , TKa 
Mtltry aodtl tptcl- 
flciilont Art 
sttondAry to tbA 
titled obJtctU** 
HSmtrt . Tht "Atttry 

Mdtl tlAltd It 

prlairy for An 
tltornttlvo objtctlvt 
"utt of hltlory At t 
tourct of K*d|ci1vt 
potter," Tblt ttetn 
a Mrt tpproprlAU 
ootl. 



Conpltlt trAntftr 
tipected lo If 
one Has Abll Ity m i, m 
om likely Has 
^hllliy Mto. 
"tttvhlng tcMArds* t 
Mill :Uely give Me 
■isttry of h Alto, 



SjluAtton* T*st prt- 
pArAtlon Mdtrtlely 
eipentlvti but tcorw- 
«Ica1 to **l up 
conditions, 
Besp onit . Mrltlen 
response rtquArts 
only pencil And pAptr 
And thut Is Inenpen- 
tlve, 

ArljfAcl. Scoring key 
dTlftcuTiy And tiptn- 
tlve to dtvtlop. 
Scoring rtlAllvtly 
iMiptntlvo, 



The question Is: Is 
It worth this titrA 
tlae And ttpente to 
gel this Accomplish- 
ment Along with 
tcconpllsh»ent A? 



*7h*3 Is an Impressionistic analysis of two testing alternatives. Judgments of Importance, generall2ab*11ty t and 
economy should be based on empirical data. Mastery Model "A" Is an author-defined objective and Mastery Model >( B" 
Is an "Ideal-defined" objective. 



the choice appears to be between economy and relevance or a matter of 
"is the relevance worth the cost?" 

Narrative of Student Use of Courseware Against Background of a 
Flowchart of the Inferred Courseware Structure * This strategy has elements 
of what we have elsewhere called " accurate, sharp, loving description" 
(Della-Piana, 1981, pp. 218*223). Me have said that the perhaps most 
useful way to illuminate the structure of courseware or how it works is 
to follow a student path through the instruction* In the present Illus- 
tration, the portrayal is in the form of a flowchart and accompanying 
narrative . The courseware used for illustration is the MECC (Minnesota 
Educational Computing Consortium) program on "Poetry," the purpose of 
which is stated as a "Joint effort between people and computer to write 
Poetry/ 1 The data juxtaposition is an author-defined path of instruction 
(inferred by the evaluator going through the instruction to see how it 
worked) in the form of a flowchart, along with a student-defined path 
of instruction (determined by collecting printouts, observing a writer 
going through the program, and interviewing the writer). The student- 
defined path is presented in narrative form and should be read with the 
flowchart (Figure 2*0) in hand* Get a copy of Figure 2*0 in hand and 
follow the flowchart as you read this much-abbreviated description of 
a student's path through the instruction* (The description is edited 
to Illustrate portrayal method, but poems used are selected poems 
produced by the student writer.) 

The student is a college graduate and an unpublished occasional 
"writer of poetry." He starts the "Poetry" program (inserts diskette, 
etc.) and types "yes" at step one (instructions desired?), whereupon 
he reads instructions in steps two and three* There are two steps 
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Q START } 



Instructions/ 
desired? 





YES 
t . 






Program purpose 


and procedure 



I 



3 Procedure 
continued 



NO 



W111 you use 
words saved 
In file? 



4a 



K&kt a flit 



m SUCH FILE 



[5a 



5b 



Type nouns 

~c — 



Typt verbs 



"Sc 

Type adjectives 



[53" 



Type Adverbs 1 



YES 




Type in 
file name 



Choose: 1) Noun 
2) Verb 3) Adj. 
4) Adv. 5) Mew 
line 6) List poem 
Continue until poem complete 



— 

vcc / Save the words \ un 

_YES f if , a f1lf? ) NO- 




( end y 




3 Another poem? 
End? 
Different 
program? 



MEM f~ 



FIGURE 2.0. Flow chart of MECC Elementary, Volume 2, Version 1, 
Menu Item #7, "Poetry." 
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for instructions because it won't all fit on one frame. The writer 
reads part of it aloud, ". . a joint effort between people and computer 
to write poetry/' He comments that "You know, you can read those 
instructions and have no idea as to what you will be required to do." 
The observer says, "You may go back and reread if you wish, or go on." 
He goes on. But before he does, he says, "I don't know about this 40 
characters per line and 48 lines, but I suppose they will remind me." 
He goes on to step four (will you use words saved in a file?). He 
says "sure" and types "yes," but the message he gets is "Type in the file 
name" and he realizes he has no file. So he types "X" and gets the 
message that the computer "can't find it" and that he has to "use new 
words." He is then instructed to type nouns (5a), verbs (5b), adjectives 
(5c), and adverbs (5d). He types in the following after a reflective 
pause: 

nouns— city, wind, garbage 

verbs- — smile, jump, speed 

adjectives- — original, strange, obsolete 

adverbs— forever, intricately, fast 
The student goes on to step six where he is asked now to write 
a poem with the computer by choosing some sequence of noun, verb, adjec- 
tive, adverb to form lines of a poem (presumably up to 48 lines and 
40 characters per line)* He decides on a four-line poem, but seems 
puzzled as to what the program is about* "What are we trying to do?" 
he says to himself. But he goes on again and types 1n the following, 
in the proper manner, signaling the computer to retrieve words (in, 
presumably, some random fashion) from each grammatical category and 
place them In sequence as Indicated* He calls for: 



line 1: noun adverb adjective noun 
line 2: adjective noun adverb verb 
line 3: adverb verb noun noun 
line 4: adjective adjective adverb 
He then types in a *6 (list poem) when asked for "new wish" and 
the following comes up on the monitor: 
city forever original wind 
intricately speed city garbage 
strange original fast 
The student commented that this was "Not a bad poem for one written so 
mechanically," and added, "but I'm not sure this teaches poetry writing." 
During a brief probing session, the writer indicated that the words he 
put in were not entirely without thought and his retrieval sequence for 
word order was selected to avoid standard-prose grammatical sentence 
patterns so that *. - it might sound more like poetry or so that something 
interesting might happen." But for this student there was something 
intriguing about the process and he decided to save the words in a file 
(step 7), naming it "Garbage," and then used the same word file for two 
other poems which came out like this; 

garbage speed garbage city forever obsolete 

wind fast Jump wind forever smile garbage 

city forever speed strange wind strange strange intricately 

speed wind garbage city 
The only comments the writer made were that both poems seemed fair 
considering the constraints, but that the last word (intricately) in the 
second poem seemed inappropriate* He noted that this process might well 
encourage experimentation with word order and might at least "generate 
ideas for poems/ 1 He also suggested that experimenting with "words put 

20 2o 



in" might be interesting and tried that the next time he chose to work 
on the program. 

In the next session he created a new word file by making a chart 
with noun, verb, adjective, adverb heading the columns and clusters of 
letters of the alphabet for rows (e.g., ab, cde, fg, hi , kl , mnt). He 
then put words in each cell as they occurred to him. The results for 
two attempts at poems using this file were discouraging to the writer- 
One "poem 11 was : 

baker live plant 
ship weekly trip head 
wave always open quick air 
truck wait miss miss 
pass wave duck kitten 
The observer asked the writer to comment on what happened. "This 
is pretty bad, pretty bad," he said, shaking his head* "Maybe I was 
lucky the first time* But I think I was really free-assodat1ng words 
to input that might fit together somehow. It wasn't really as random 
as I did for the last two poems* It was something Hke this: 

nouns: citj£ (let's see what goes on 1n the city), wind (I'm 
thinking of Chicago) and garbage (remember when the 
garbage collectors were on strike? Mas that Chicago 
or New York?) 

verbs: smile (let's try to be happy anyhow], and jump and 
speecj (some active things) 

adjectives: I don't Hke adjectives. But let's see, original 
(Hke strawberries with seeds on the outside), 
strange (I guess original is also strange, but a 
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different connotation) andt lets see, obsolete 
(garbage should be obsolete. It's certainly strange 
and gets mighty original at times), 
adverbs: I like adverbs less than adjectives. Let's see. 

they are tied to verbs. Forever (smile forever) 
and fast (fast jump, I don't know. Speed made me 
think of fast. That's great. Speed is a noun too). 
Intricately (I like that word; let's modify all 
verbs with intricately. Life is complex,). 
The writer was worn out and stopped the session here. 

The next day the writer made a sixth try . He decided to take 
words from a poem by e. e, cummings and didSQ, putting them down in 
columns and then putting them 1n a word file as follows: 



Noun Verb Adjective Adverb 



spring 


is 


perhaps 


carefully 


hand 


comes 


strange 




nowhere 


arranging 


known 




wi ndow 


look 


like 




people 


stare 


new 




thing 


changing 


old 




everything 


placing 






fraction 


breaking 






flower 








Inch 








air 








anything 




i 


i 



The poem is, of course, "Spring is like a perhaps hand." The word 
order requested by the writer for retrieval in step six was: 
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noun 


verb 


adjective 


noun 


adverb 


verb 


adjective 


noun 


adverb 


verb 


adjective 


noun 




vcr u 






noun 


verb 






noun 


verb 






noun 


noun 


verb 




noun 


noun 


verb 




noun 


verb 


noun 





The poem was then asked for. After reading it over and saying 
11 It has possibilities; I think I'll change It a bit," he made some 
changes* Here is the initial poem* followed by the revised one. 



air 


stare 


perhaps 


fraction 


careful ly 


1s 


like 


everything 


carefully 


arranging 


like 


spring 


flower 


stare 






peopl e 


stare 






hand 


breaking 






air 


peopl e 


stare 




fraction 


anything 


look 




everything 


breaking 


anything 


fl ower 



The revision was extensive: 

air Is a perhaps fraction of stone 

carefully put is everything 

careful ly arranging 

flower and stone 

people and stone 

air people flower stone 

a fraction of anything — look 

anything makes everything 

The writer commented, ''It looks like having some words that 
'go together' will at least create an Idea for a poem. These aren't 
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great poems. They are, in fact pretty bad. But if you experiment with 
putting them in the file in different ways* and then experiment pulling 
them out in different patterns, you might get some good notions. And 
then break away from the computer and write. It has possibilities. 
But I'd like to be able to change the program. I'd put in more word 
types. Then I'd put thum 1n so I could retrieve them in more than one 
way. Like, retrieve by Initial letter, part of speech, synonym or 
antonym; and connotation of good or bed, movement or passivity, or other 
such things. I've had enough though. The program would bomb with anyone 
who didn't know much about writing poetry. But it got me thinking. I 
won't forget It for a while. I doubt I would want to use 1t again* 
But 1t got me thinking. I'll try out some new things*" 

Brief Summary Flowcharts of Two Persons Going Through a Program 
and a Narrative for One . This portrayal strategy 1s based 1n part on 
Auerbach's method of formal analysis outlined In Della-P1ana (1981, 
pp. 223-228). The form this takes in the current Illustration is a 
summary flow chart with three representations of pa*hs through the 
"Poetry" program described above (see Figure 3.0). The center flowchart 
1s our own simplification of the inferred author-designated path through 
4 the program. The flowchart on the left Is a simplification of the path 
of an occasional writer of poetry and the one on the right Is the path 
of an Inexperienced and reluctant writer. Thus, in a summary fashion we 
have mul tipersonal representations of consciousness with three different 
perspectives as to what is involved in "going through the program." Let 
us call the occasional writer WRITER, and the other RELUCTANT, using 
caps in both cases. We follow primarily the thoughts of RELUCTANT, with 
some reference to WRITER and Gome to an observer. 
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Path of Occasional WRITE R 
of Poetry {Unpublished!" 



Simplified Representation of 
Inferred Author-Designed Path 



Path of Inexperienced RELUCTANT 
Writer of Poetry 



Hikes flii of words 
using i conscious 
tethod. 



Makas new file of 
words using dl f far- 
ant wit hod. 



Rms new file of 
words using dlffer- 
ant tethod* 



Stops, Talks about 
how to sodl fy pro- 
grat to be more 
useful. Talks about 
IdetS for writing 
obtained frot try- 
out of the prograt* 



Writes a poet, Savas 
words In a file and 
writes several poet* 
exper (tenting with 
word order. 



Writes a poet* savas 
words In a Mia and 
writes several poets 
axperttent Ing with 
word order • 
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Mrl tes a poen, Saves 
words In a file and 
writes several poets 
eKper Iran ting with 
word order. 



SAHUE POEH 

city forever original wind 
Intricately speed ctty garbage 
strange original fast 



Start 



Instructions 



I-*— 


Ulll you use words 
saved In a Mia? 


Type in words to aaka 
a flla; nouns * varfc* 
adjectives* adverbs. 




ves 

* 




Type In rlla name. 





Type word ordar tsoM 
combination of noun* 
varb» adjectlva* id- 
varfaj for lines of a 
until completa, 

i - 



List (MC«. 



ves 


Save words In a file? 


I 


HO 




Another poet? Cnd? 
01 f ferent pro gr at 7 



Menu 



Nakas a fHa of 

words using what- 
avar cotes to tlnd 
for Input of noun* 
Varb* adjectlva* 
advarb. 



Uses words to wrlta 
one potA with no 
conscious design for 
wOrd order. 
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Dots not 
In a rlla 


save words 






Stops after one 
poet. Does not uSa 
old file again nor 
take a new one* nor 
wants to* 



SAMPLE POCH 

Bob fly big 
bulb run slowly 
bulb Super jutp 
run bulb bulb cowardly 



FIGURE 3.0, Flowchart of simplified! Inferred* author -designed path through HECC 
paths , 



'Poetry** and two student 
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RELUCTANT was in the Ninth Grade at the time he tried out the 
Poetry program. He had above-average performance levels ;in math and 
reading, had never written a poem In his life nor wanted to. After 
getting the diskette going and moving through the initial instructions, 
he came upon the step, "will you use words saved in a file?" He paused, 
read the instructions aloud with an incredulous emphasis on "in a file." 
What does that mean, he thought aloud as he fumbled his way into the 
Instructions to type in words for ncuns, verbs, adjectives and adverbs. 
"And nouns," he thought, "is it o.k. to put in a person, place or thing? 11 
He put in 80B-80ST0N-8UL8 and smiled as he noted how hard it is to 
'think up 1 a noun and how much easier to label one if someone shows you 
a list of, say, nouns and verbs. So much for nouns; at least there are 
some in there now* Verbs* Verbs* He turned to the observer, inquiring 
whether verbs describe nouns* "Mot that it mattered," he thought to 
himself* His head moved back and forth as if watching a ping-pong game; 
first puzzling over the meaning of what was on the monitor, and then 
muttering a question or answer while looking to the observer for confirm- 
ation* He gave a sign when finished with inputs and another one when 
confronted with the puzzle of choosing words by grammatical type and 
ordering them into lines and verses. "Do I list the poem," he thought, 
before he even specified the word sequences. He typed a sequence of 
numbers (1,2,1,5) which represented "noun, verb, noun, list poem," and 
got the feedback "that is not a number*" When he finally got the poem, 
it vas 

Sob fly big 

bulb run slowly 

bulb super jump 

run bulb bulb cowardly 
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He commented iPon hew he would rather not do ft again and thought 
to himself how painful and useless 1t was and made moves to leave. The 
observer was curious about how another poem might turn out and twisted 
RELUCTANT 1 s arm, RELUCTANT went through the entire painful process 
again and came out with a decidedly discouraging poem. To get RELUCTANT 
to make another try. the observer told RELUCTANT about WRITER and how 
writer tried different ways of putting words in and different sequences 
for pulling them out and showed him some of the poems, RELUCTANT listened 
and sometimes saw the observer's Hps moving, but strangely heard 
nothing; other times he heard the words the observer uttered and they 
matched the observer's Hp movements all right, but the words seemed to 
go over his head, or through him or off of him, off the walls, off 
anything in the way, or dissolve Into the air. No, he wasn't going to 
try 1t again no matter how successful or engaging the experience was 
for WRITER nor how persuasive the observer. 

Comments on Interpretation * The three portrayals Illustrated 
above are only a sample of ways in which data may be presented consistent 
with the perspective and strategy we have outlined* We intend to explore 
other strategies, Including hypothetical dialogue (between student and 
program developer) and exploratory data analysis. It should be apparent 
from the Illustrative portrayals that we have not made the final judgment 
for the user. Our hope 1s that we have done more* Our Intent was to 
portray certain data juxtapolsltons in ways that involve the reader in 
wanting to make interpretations and to make the courseware transparent 
so the reader can make Interpretations and judgments consistent with 
Intended uses of the courseware. But all this is our own Intent. In 
the next section we propose how we plan to investigate the actual effects 
of this kind of portrayal* 
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7.0 Investigation of Portrayal Effects 

Me have specified some strategy and tactics for gathering in-depth 
data on courseware packages and for portrayal of that data in alternative 
forms that might have the effect of Involving audiences in making their 
own interpretations and judgments about the courseware for their own 
intended use. We have suggested that for the buyer the use would be 
to guide selection of courseware to "try out. 11 For the teacher managing 
students using courseware, the purpose would be to guide adaptation of 
the courseware or proposed utilization procedures for use within their 
own context. For the designer/developer the purpose might be to motivate 
redesign and redevelopment or to motivate modifications of representations 
concerning courseware. Claims or hopes for such effects of portrayals 
of courseware criticism must be coupled with Investigatons. What might 
be the form of such investigations? 

We do not outline methodology for such studies here, but there 
are some dimensions of the investigations that are worth noting. 

The coursework studied should be a package that has been initially 
screened and found to be educationally valuable, technically competent, 
thorough in documentation, and widely used. In addition, the publisher 
should allow printout of student input for purposes of the investigation, 
subject to appropriate protection of copyright. 

The setting of the study should be one where the program 1s in 
operation at a stage where most procedural concerns are worked out. The 
institution naturally should work out a detailed agreement that protects 
the rights of students and others and clearly specifies what will be 
done and the rights and responsibilities of all parties. 

26 



The specific design would have to be worked out given the context 
and the decisions of most concern within that context. Thus, the student 
population sampled would have to be representative of the range of students 
for whom the program 1s targeted both by the publisher and the local user. 
The depth of data gathering would have to be a function of both resources 
and questions of concern. At any rate, one would end up with data 
appropriate to the decisions of concern within Table 2*0 or others added 
by the study group. 

Alternative portrayals would be prepared in ways that are designed 
to achieve the Intended purposes. In the case of teachers, one would 
begin with a systems model of teacher decisions with the context of 
Intended use. Then alternative portrayals of the courseware criticism 
would be directed toward utility for those decisions and tested against 
actual teacher responses in an appropriate evaluation study* For the 
developer* one would begin with a systems model for design and develop* 
ment and prepare alternative portrayals of the courseware criticism 
directed toward utility for development decisions* Again the alterna- 
tive portrayals would be tested against actual developer decisions on 
design or redevelopment using an appropriate evaluation study. 

For an excellent suranary of approaches to the study of utilization 

of evaluation reports, see Weiss (1982). After a consideration of 

t 

conceptualization of use* Weiss cites a variety of studies in which 
observation* interview* the tracking of documents* and participant- 
observation methodological strategies were used to answer questions 
concerning effects of evaluations on people* resolution of issues* and 
organizational behavior. 
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8*0 Courseware Criticism Checklist 

We noted at the end of Section 4.0 that we would propose an 
alternative to the in-depth analysis outlined in this paper. The alter- 
native Is an open-ended checklist that goes beyond the simplicity of 
the usual rating sheets t but not as far as the in-depth procedures we 
have outlined above* We have completed the checklist only for "student 
path*' 1 Our intent is to eventually develop a checklist for the major 
data sources noted in Figure 1*0. The use of this checklist will 
require training and provision of many examples of its use in different 
contexts . 

In Appendix B we present the "open-ended checklist*" On the 
following pages we present one illustration of the kind of data that may 
be gathered with the checklist* The program used for the tryout is 
'Arithmetic Racing*" a drill and practice program in arithmetic funda- 
mentals published by Math Software* 1980* The form of our portrayal 
is simply the checklist itself with observations and student responses 
written in* This is certainly one useful portrayal method* although we 
believe it would be important to experiment with alternative portrayals 
such as* for example* a hypothetical dialogue between the student and 
the developer using the open-ended checklist data plus program documen- 
tation. Following the illustrative example of the open-ended checklist 
data* we have included the Northwest Regional Educational Laboratory 
Microsift evaluation of the same program to illustrate the typical 
checklist approach* 
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Sample Completed Checklist Based on Teacher Observation 
of one Student in the Program 

MICROCOMPUTER COURSEWARE EVALUATION OPEN CHECKLISr 

Student: Grade 5,5^ male. Accurate, in basic operations but low test scores 

on speeded tests. 

Instructions : Where a yes/no 1s asked for, circle your response. For 
open-ended questions, write in your response in the space provided. Be 
brief, descriptive, and to the point. Use typical examples of student 
performance where appropriate. 



1. Can the student start up the program? 




If not, what skills or additional Information must the student have 
in order to start? 

Student must have knowledge about "how to start the system" and/ or 
to "boot" the system if diskette and hardware are DOS incompatible 



2* Can the student proceed through the program with the Information 
provi ded ? Yes No Somewhat 
somewhat 

a. If by what means does the student do it? 

displayed Instructions 

manual /workbook 

other a3 ked teacher to clarify what to do 



b. If no, what skills or additional information must the student 
have? 



3. Does the student have a Yes/No option \f£r instructions at the 
beginning of the program? Yes Qio^ 

4, What does the student actually do when beginning instructions are 
unclear? 

Asked teacher "what do I do?" 



5. Does thestudent have the option to go back to the instructions? 
Yes Q§) 

If no, how does the student proceed in order to get the necessary 
information to go on? When teacher was asked "what do I do? f ^ she 

said "What do you think you should do?" student Just went on 

through the program. 
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Can the student control the path of instruction? ^Yes) No 
a. In which way? 

skip sections 

go back 

x increase or decrease speed select Levels (l=mich time and o^little 

time) of time allowed for answer. 
x level of difficulty "largest number to operate on" 

x other choice of arithmetic operations 



b» Ooes the student know what the consequences are for each choice? 

Yes No Not at first. But by the end of three S-minute 
sessions student grasps alt options. 
c. What were the student's reasons for making the choices he/she 
did make? Chose multiplication "because I like it and I'm good 
at it*" Avoided subtraction and division "cause they are 
harder." Chose high speeds "cause I'm good" but lowered 
to Level three when in difficulty. 

Can the student input responses called for? . Yes No 

What kind of responses are asked for? 

{TT\ Numerical; what is the input order? ^left to rjqKt) 
^— ^ Right to left 

Inconsistent —no. 

b* Alphanumeric/verbal? One letter One word Phrase 

Is word order and spelling important for correct evaluation 
of student input? Yes No Does not apply. 

What does the student do when the instructions to do something 
are not clear? For example, how does the student attempt to get 
clarification or otherwise react to the lack of clarity? Asks 
teacher or thinks out loud with comments or rhetorical questions 
like "I wonder how they figure points?" or "what do you mean 
r watch my speed** I was fast and got it wrong. 0h? f ' 



9* Can the student correct an input before the computer evaluates 

the input? Qf es^ No Only on speed level /nowever^ This student 
hung system by pressing reset and trying to exit during the drill* 
10* Is the student correction of iriflut errors aided by provision of 
program messages? Yes Qlo^ 

11* How are correct responses acknowledged or confirmed? 

a. x graphics 

x exclamatory remarks 
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11 * a. (continued) 



x sound 

x continuation of program 

b. Describe any special features of the confirmation* Sound feedback 
(a "charge** song)> exclamatory remark "Yes NAME Keep Racing!"^ score, after 
20 problems: flags portrayed in different color rectangles like racing flags 3 
the remark "charge arithmetic racer" and then score and bonus points. 

c. What is the effect of the confirmation on the student? 

student mimicked the rf charge ff song 



12* How are incorrect responses acknowledged or corrected? 

a* graphics 

* exclamatory remarks 
x sound 

x continuation of program 

b. If error recognition and correction messages are so general 
that they miss important features of the student's errors, 
give an example of the student error and the acknowledgement 
and correction message* Student commented out loud tf I added 
instead of multiplying" or "I hit the wrong key. I was thinking the right 
number/* Feedback was "watch your speed NAME. The answer is x * 



Describe any other special features of acknowledgement or 
correction of errors* 

Program continues to next problem* 



d* What is the student's response to the acknowledgement of error? 

rt that is what I put in." or "oh, is that the answer?" or "I'm not 
very good at this.*' or rr I always get confused. 

To the error correction process? 

"I'll have to try harder" or "I'll have to remember the answer." 



To error messages that do not discriminate his/her specific 

error? "Wait a minute. My answer was wrong y it wasn't slow. Why does it 

say "watch your speed." Oh> I see. That is clever. If I go too 

fast and get it wrong 3 it says "watch your speed." If I go too slaw 

and get it right 3 it still says "watch your speed." 
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13. What student accomplishments (beyond the single response) are 
singled out for "reward" or special commendation through program 
feedback? See #11* After every 10 problems you get flags and stuff and 

you can raise your points by choosuig harder problems (higher numbers), 
more operations* or higher speeds. But these are not noted specifically* 
You figure that out or read it in the manual* 

What criteria are used for "successful accomplishment?" 

correct answer 
under speed limit 
selection of larger numbers 
choice of arithmetic operation 

14. How does the student respond to special commendation or reward 
systems of the program? 

This student Just mimicked the "charge song*' or said 
"I'm good at this. " 



15* Qaes the student have an option to repeat the program or end? 
Qes) Mo 

a. If the student attempts to repeat, what happens? 

goes back to selecting another arithmetic operation (or same one), 
largest number desired* and the speed level for making a 
response. 

How does the student react? See # $ This stu d ent chose 

multiplication most y started at high speeds and levered them 3 
and slowly made numbers higher (under coaxing from teacher)- 



b. If the student attempts to end, what happens? 

gets blank screen with flashing cursor. 

How does the student react? 

Asks "what do I do nau?" 

16- When the student completes the session, what does he/she say in 
response to the following questions: 

a, What was the lesson about? Tell me more. 

Student laughed and said y "Doing arithmetic. Like multiplication 
and addition and stuff. " 
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16* (continued) 

b. Do you think this kind of work will help you? How? 

"It would if I practiced what I needy like subtraction and division* 
But I like multiplications so I did it most." 

c* What were the difficulties you encountered in going through 
the lessons? 

"My fingers kit the wrong keys cause I was trying to go fast* 
And I got tired of the title and address ond stuff every 
time you start over*" 

d* Would you recommend this program to other students? Yeah* 
Who would they be? >* My friends*" 



Why? "So they can play the games after," Anything else? 
"Oh, they can do arithmetic." 



Why not? 



35 



9 

:RLC 



microTsiFT COURSEWARE EVALUATION 



Arithmetic Racing 



PRODUCER: Math Software 
1233 BUckthorne Place 
Deerfietd, IL 60013 

LOCAL DISTRIBUTORS* Contact producer for list 

EVALUATION COMPLETED* Fall 1981; Revised 
2/1/82 

VERSION*© 1980 

COST* Hot sold individually? sold in packages of 5 
to 10 programs ranging from $100 to $250 

ABILITY LEVEL; Grades 4-11 

SUBJECT* Mat hematics* speed and accuracy 

<frQl of arithmetic operations 
MEDIUM OF TRANSFER* 5" flexible disk 
REQUIRED HARDWARE* 32K Apple 0 or 0 Plus, 

one disk dive, monitor 
REQUIRED SOFTWARE* DOS 3.2 or 3*3, 

Applesoft 

INSTRUCTIONAL PURPOSE* Standard 

instruction, enrichment 
INSTRUCTIONAL TECHNIQUE* Game, problem 

solving, dritt and practice 
DOCUMENTATION AVAILABLE* Suggested 

grade level, program operating instructions, 

demonstration 

INSTRUCTIONAL OBJECTIVES* To Improve 
students' arithmetic skills in addition, 
multiplication p n£ tract ion, and division; to 
provide an interesting interactive environment for 



remediation work in arithmetic facts; and to 
develop speed and accuracy in working basic 
arithmetic operations. 

INSTRUCTIONAL PREREQUISITES* The program 
assumes that students (enow basic arithmetic facts 
concerning the operations of addition, 
multiplication, subtraction and division. Students 
also need to understand the rules governing the 
operation of the computer game* 

CONTENT AND STRUCTURE* ARITHMETIC 
RACING is a game of timed arithmetic practice 
for students Grades 4-11. Players first select 
addition, subtraction, multiplication, or division 
and then specify the largest number they want the 
computer to give them. Pleven also select a 
speed level from 1*5. The computer then assigns 
a point value to each problem based on these 
selections. * A 25 point bonus is added to the score 
for answering each of the ten questions correctly. 

ESTIMATED STUDENT TIME REQUIRED* Not 
available 

POTENTIAL USES: The program may be used in i 
classroom setting to provide drill and practice in 
basic arithmetic operations. 

MAJOR STRENGTHS* The game format offers an 
incentive for students needing drill and practice 
in arithmetic operations. 

MAJOR WEAKNESSES* None e *ted. 



EVALUATION SUMMARY 
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There must be some word or two to provide a kind of closure or 
antlclosure for a document such as this. It seems to us that the state of 
the art 1n courseware criticism calls for antlclosure* 

Paul Heckel writes that "There may be disagreements on who the 
typical user for a product 1s, but the best designs are those created 
by people who have a clear and realistic picture of the ultimate user" 
(Meckel, 1982, p. 12). In a series of articles on "intelligence design" 
or "software for people" 1n InfoWovld beginning with the July 19, 1982, 
Issue, Heckel keeps this "user orientation" alive as he considers the 
design of communications with a concern for visual processes of the user, 
empathy with the user, user control, helping the user to cope, leading 
the user without a manual, making complex design appear simple, and 
so on. The emphasis is always on the user; "The organization that Is 
Important 1s not that which appears on the screen, but that which is 
created 1n the user's mind" (Heckel, 1982a, p, 27), As with Meckel's 
design of courseware, so may 1t be with courseware criticism. It 1s 
Informed by the user and portrayed for the user. 
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APPENDIX A 



STRUCTURED OBSERVATION OF STUDENT PATH IN GOING 
THROUGH MICROCOMPUTER COURSEWARE 



Materials to Have Ready 

* Computers In working order, preferably with print capability for 
collecting student responses, 

* Diskette(s) for review, 

* Workbook or other hard copy material if available. Two copies, 
one for student(s) and one for an "observer," 

* Stopwatch to time segments of the work, 

- Extra paper for use If needed by student and notebook for 
"observer," 

- Tape recorder to pick up student's thinking when "thinking aloud" 
and at probing points, 

* Pens for student and observer. Do not use pencils since a complete 
record with no erasures and no crossouts is desired, 

Tryout Space Arranged 

* Quiet area free from interruptions or observers, 

* Comfortable chair for student to work at computer with space and 
lighting for writing 1n workbook, 

* Chair for observer next to student so that student's work can 
be observed on the computer and In the workbook, 

* Writing surface of observer. 
Guidelines for Activity During the Tryout: 
Explain to Student the Purpose of the Tryout 

* It is to test the material, not the student, 

- Me wish to find out how people use the material and what they 
find most useful or interesting, 

* Me wish to know where 1t Is easy and where it is difficult so 
that we can find better ways of using the material to make 1t 
easier for others . 

Ask Student to Make Responses Orally and in Writing So That You Can 
Follow His/Her Thinking 

* "Think and talk out loud" as much as possible, 

* Write down your answer compl etely where possible or necessary. 
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• Do not scratch out anything written so that it cannot be read. 

• Work at a comfortable pace, but slow enough so that the observer 
can follow. 

• As much as possible, write in the nctebook or scratch paper in 
sequence so we can tell the order in which the work was done. 

Tell the Student What You (TA) Will Be Doing 

• Collecting print copy of student responses if possible { ,r so I can 
go back to specific places and ask you what you did or thought at 
that point 11 }, 

• Taking notes ("so I can go back to them to see how people use the 
materials' 1 ), 

• Observing ("to see how it works, and to mark down where it is 
easy, where difficult"). 

• Timing ("to see how long different segments take. But don't try 
to rush 1t, We want to see how long It takes at an easy pace."), 

• Taping("so I can listen to 1t and follow your thinking' 1 ). 

• Probing or questioning ("so I can get a clearer picture of your 
thinking at places where 1t is difficult . or more interesting"). 

Respond to Student When Asked "What Do I Do Now (Next)?" or "What Does 
This Mean?* At Any Time During the Work 

• Say "do what you think 1t asks you to do*" 

• Say "rea 1 what 1t says (on the monitor/in the workbook) again and 
see 1f : makes sense. " 

• If necessary, write out additional Instructions in the student's 
book to help make 1t clearer, or use notepaper, but keep a record 
of what you wrote since this is part of the course now. If you 
do say something orally, write 1t down also. 

Structured Interview Following Completion of Courseware Segment 

Probing Points: Probe at Points Marked by You As Student Goes Through 
the Course 

• "Error" points (where student response 1s not the "keyed response"), 

• Long latency points (where there was a long delay 1n student 
response or the student Indicated something had gone awry or that 
something was being deliberated). 

• Ambiguity points (where student comment indicated unclear or 
ambigulous material, whether the implication was a "material 
deficiency" or "personal deficiency"), 

• Appropriateness points (where student comments on how the course 
"should be" Ideally, with or without specific criterion evidence 
to support the judgment). 
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• High-utili ty points (wherestudent comments on particular interest, 
usefulness or value of the course or activity). End on this 
positive kind of probe. 

Probing Procedure (preferably after each course segment ) 

• Refer to or point to or bring back on the screen or turn to work- 
book page or printout where probe 1s focused, 

• Say something like n I noticed you (did, said, etc.) at this point," 

• Ask open-ended questions about the point first (could you tell me 
more about what you were thinking at this point, what went through 
your head, what ideas you had or steps you went through). 

• Ask for 'anything else" to get the student to pursue further his/her 
thought processes* 

• If the student is upset, use an appropriate "supportive" and 
"accepting" style that is honest. 

• Probe for specific questions 1f any are left after the open-ended 
probe (e.g., "so you weren't sure if it meant 'this' or 'that' — 
can you tell me how you thought about the choice and how it came 
out this way?"). 

Summation Probes or Exit Interview After Completion of Diskette and 
Related Hard Copy Material 

• What were the major obstacles or problems in going through the 
lessons? \ 

• Would you recommend it to other students? Why? Why not? Any use 
for it that you can see? 

• Do you have any other comments about the course and about my 
observing and making notes this way? 

• Thank the student appropriately and sincerely comment on special 
features of the student's work you found most pleasing or 
commendabl e. 
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APPENOIX B 



MICROCOMPUTER COURSEWARE EVALUATION 
OPEN-ENOED CHECKLIST 



Instructions : This checklist requires observation of a student going 
through courseware Instruction along with occasional interview or 
questioning of the student. 



• Prior to beginning the observation-Interview, go through the 
program or part of 1t yourself to get acquainted with the 
procedure, 

• Use this checklist to note your own responses In using the course, 

• Select a student typical of the population for whom the course 
was designed, 

• Explain to the student that you are check out the computer and 
the course so that you will know more about how it works, 

• Tell the student you would like him/her to go through the 
program "thinking out loud" as he/she goes along so that you can 
get an idea of what 1s going on in his/her head, 

• Demonstrate the "thinking out loud 11 by using a different program, 
and then have the student use that program to practice "thinking 
out loud" as he/she goes along, 

• Have an open-ended checklist available with space between Items 
to allow writing in observations. Use a cassette tape recorder 
1f you can or even a videotape recorder If possible. Use a 
clipboard for writing. 
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MICROCOMPUTER COURSEWARE EVALUATION OPEN CHECKLIST" 



Instructions : Where a yes/no Is asked fori circle your response. For 
open-ended questions! write in your response in the space provided. Be 
brief, descriptive! and to the point. Use typical examples of student 
performance where appropriate. 

1. Can the student start up the program? Yes No 

If noti what skills or additional information must the student have 
in order to start? 



2. Can the student proceed through the program with the information 
provided? Yes No 

a* If yes, by what means does the student do it? 

displayed instructions 

manual /workbook 

other 

b. If noi what skills or additional information must the student 
have? 



3. Ooes the student have a Yes/No option for instructions at the 
beginning of the program? Yes No 

4. What does the student actually do when beginning instructions are 
unclear? 



5. Ooes the student have the option to go back to the Instructions? 
Yes No 

If no, how does the student proceed in order to get the necessary 
information to go on? 




6. Can the student control the path of instruction? Yes No 
a. In which way? 

skip sections 

90 back 

^ increase or decrease speed 

level of difficulty 

other 



b. Does the student know what the consequences are for each choice? 
Yes No 

c. What were the student's reasons for making the choices he/she 
did make? 



7, Can the student Input responses called for? 
What kind of responses are asked for? 



Yes 



No 



a. Numerical; what 1s the input order? Left to right 

Right to left 
Inconsistent 



b+ Alphanumeric/verbal? One letter 



One word 



Phrase 



Is word order and spelling Important for correct evaluation 
of student Input? Yes No 

8* What does the student do when the instructions to do something 
are not clear? For example, how does the student attempt to get 
cl irif1cat1on or otherwise react to the lack of clarity? 



9, Can the student correct an Input before the computer evaluates 
the Input? Yes No 

10. Is the student correction of Input errors aided by provision of 
program messages? Yes No 

11* How are correct responses acknowledged or confirmed? 

a. graphics 

exclamatory remarks 
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11 . a. (continued) 
sound 

continuation of program 

b. Describe any special features of the confirmation. 



c. What 1s the effect of the confirmation on the student? 



12. How are Incorrect responses acknowledged or corrected? 

a. graphics 

exclamatory remarks 

sound 

continuation of program 

b. If error recognition and correction messages are so general 
that they miss Important features of the student's errors, 
give an example of the student error and the acknowledgement 
and correction message. 



c. Describe any other special features of acknowledgement or 
correction of errors. 



d. What 1s the student's response to the acknowledgement of error? 



To the error correction process? 



To error messages that do not discriminate his/her specific 
error? 




13* What student accomplishments (beyond the single response) are 
singled out for "reward 1 * or special commendation through program 
feedback? 



What criteria are used for "successful accomplishment?" 



14, How does the student respond to special commendation or reward 
systems of the program? 



15* Ooes the student have an option to repeat the program or end? 
Yes Mo 

a* If the student attempts to repeat, what happens? 



How does the student react? 



b+ If the student attempts to end, what happens? 



How does the student react? 



16, When the student completes the session, what does he/she say 1n 
response to the following questions; 

a. What was the lesson about? Tell me more* 
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16. (continued) 

b. Do you think this kind of work will help you? How? 



c. What were the difficulties you encountered 1n going through 
the lessons? 

d. Mould you recommend this program to other students? 
Mho would they be? 

Why? 

Why not? 



46 



ERLC 




LIST OF REFERENCES 



Bronfenbrenner, U. The Ecology of Human Develovjnent. Cambridge: Harvard 
University Press, 1979. 

Calfee, R. "Cognitive Psychology and Educational Practice," In Review 
of Research in* Education, David C, Berliner, Ed. Washington, D.C.; 
American Educational Research Association (19B1), pp. 3-73. 

Della-Piana, G. "Literary and Film Criticism," in Metaphors for 

Evaluation: Sources of Neu Methods, Nick L. Smith, Ed. Beverly Hills; 
Sage (19B1), pp. 211*246. 

Della-Piana, G. "Film Criticism," in New Teohniaues for Evaluation, 
Nick L. Smith, Ed. Beverly Hills: Sage (1981), pp. 274-285. 

Della-P1ana, G. "Film Criticism and Micro- Computer Courseware Evaluation," 

in Field Assessments of Innovative Evaluation Methods, Nick L. Smith, 
Ed. San Francisco; Jossey Bass (19B2), pp. 11-2B. 

Gilbert, T, F. "Saying What a Subject Matter. Is, 1 * Instructional Science, 5 
(1976), pp. 29-53. 

Gilberti T. F. Human Competence: Enaineering Worthy Performance. 
New York; McGraw-Hill, 197B. 

Glaseri R. "Instructional Psychology; Past, Present and Future. 1 * 
American Psychologist, Z?(Z) (19B2), pp. 292-305. 

Guba, E. , and Lincoln, Y. S. Effective Evaluation: Improving the Useful- 
ness of Evaluation Results Through Responsive and Naturalistic 
Approaches. San Francisco; Jossey-Bass, Inc., 19B1. 

Heckel , P. "Maxim for 'Intelligence Design 1 ; Know Your Users." Info 
World, 4(28) (July 19, 19B2),pp. 11, 12. 

Heckel, P. "Communicate in Specifics." Info World, 4(Z4) (August 30, 
19B2), pp, 26, 27. 

Resnick, L. B. "Instructional Psychology," Annual ^evieu of Psychology, 
32 (19B1), pp. 659-704. 

Simon, H. A. "Designing Organizations for an Information-Rich World," 

in Computers, Communications, and the Public Interest, M, Green- 
berger, Ed, Baltimore; Johns Hopkins University Press, 1971. 

Smith, N. L. , Ed. Commmication Strategies in Evaluation* Beverly Hills: 
Sage, 19B2. 



47 



Weiss, C. H. "Measuring the Use of Evaluation," 1n Evaluation studies 
•ievisw Annual, ?, E. R. House, Ed. Beverlv mils: Sage (1982), 
pp. 129-145. 



48 5; 



